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Abstract 


As postsecondary institutions increasingly integrate technology into developmental 
education, it becomes important to understand how technology is used in these programs, 
what challenges institutions have encountered relating to the technology, and what 
considerations institutional leaders take into account when deciding whether and how to 
integrate technology in developmental education. This study explores these questions 
drawing from semi-structured interviews with key personnel from 31 open-access two- 
year public colleges, 11 broad-access four-year public colleges, and 41 state-level 
organizations overseeing such colleges. We find that institutions are integrating a variety 
of instructional, course management, and student support technologies into 
developmental education. In doing so, institutions have encountered a number of 
challenges, particularly with regard to end-user difficulties with technology. 

We also find that evidence of effectiveness of technology for improving educational 
outcomes was considered by a number of organizations in our sample when making 
decisions about technology use in developmental education; however, other 
considerations — particularly those based on costs and resources — were also quite 
influential. Indeed, such economic considerations were described to us more often than 
evidence of effectiveness by respondents discussing reasons for using technology in 
developmental education. 




Contents 


1. Introduction 1 

2. Background: Developmental Education and Technology-Centered Reforms 3 

3. Methodology 5 

Data Sources and Data Collection 5 

Data Analysis 7 

4. Results 9 

Technology Use in Developmental Education 9 

Reasons for Organizational Decision-Making About Using Technology 16 

5. Discussion and Conclusion 24 

References 27 




1. Introduction 


As institutions of higher education look for innovative and more effective ways 
to deliver course content and to connect more broadly with students, campus decision 
makers are increasingly turning to a variety of technology-based options. These options 
range from new computer labs, to software-based homework, to courses or degrees 
offered entirely online (Bell & Federman, 2013; Epper & Baker, 2009; Jacobson, 2006; 
Twigg, 2003; Zachry & Schneider, 2010). Developmental education programs — 
designed for individuals who begin postsecondary education demonstrating a lack of 
college-level skills in subjects such as English or mathematics — similarly make use of 
technology to help students transition into college-level coursework. For example, math 
and reading instructional software products have been used in developmental education 
programs to assess students’ individual academic weaknesses and to provide targeted 
instruction and testing to help students master those skills (Epper & Baker, 2009). 
However, analyses show mixed results regarding the relationship between content-driven 
software and student outcomes (see, e.g., Raines & Clark, 2016), and little is known about 
how higher education leaders make decisions about what technology to use in 
developmental education. Understanding how leaders make these decisions is important 
for determining whether considerations of effectiveness for student learning are being 
prioritized, as well as for developing strategies for the implementation of educational 
technologies. 

Using semi-structured interviews conducted by the Center for the Analysis of 
Postsecondary Readiness (CAPR), this study explores the ways in which technology has 
been used in developmental education, the challenges that using such technology has 
presented for institutions, and the considerations underlying decisions about how and why 
to use technology. We find that institutions are integrating technology into developmental 
education in a variety of ways, which can be categorized into instructional, course 
management, and student support technologies. Institutions have encountered a number 
of challenges while implementing technology in developmental education programs, 
particularly with regard to end-user difficulties with technology. We also find that 
evidence of effectiveness of technology for improving educational outcomes was 
considered by a number of organizations in our sample when making decisions about 
technology use in developmental education; however, other considerations — 
particularly those based on costs and resources — were also quite influential. Indeed, 
such economic considerations were described to us more often than considerations of 
effectiveness by respondents discussing reasons for using technology in developmental 
education. The results of this analysis illuminate institutions’ experiences with 
technology in developmental education as well as the extent to which considerations of 
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effectiveness for improving educational outcomes is prioritized when colleges make 
decisions about instructional and student support technologies. 
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2. Background: Developmental Education and 
Technology-Centered Reforms 


Developmental education programs provide instruction and academic support for 
students who enter college lacking assessed college-readiness skills in at least one 
academic area (typically mathematics, reading, or writing), for the purpose of enabling 
students to develop the skills needed to meet requirements for college-level coursework 
(Bailey & Cho, 2010; Boylan & Bonham, 2007; Jaggars, Edgecombe, & Stacey, 2014). 
Students are typically referred to a single course or to a sequence of two or three 
developmental courses in a single academic area. While nearly 60 percent of new 
community college students place into developmental education (Bailey, 2009; Bailey & 
Cho, 2010; MDRC, 2013), these students often straggle to succeed. Approximately 30 
percent of students in developmental reading and in developmental writing do not pass 
all their developmental courses in these areas, and about 70 percent of students do not 
pass all their developmental mathematics courses (Attewell, Lavin, Domina, & Levey, 
2006, as cited in Bailey, 2009). Meanwhile, recent estimated annual costs for providing 
postsecondary developmental education in the United States total more than $3.5 billion 
(Bettinger, Boatman, & Long, 2013). 

It is within this context that calls for developmental education reform have been 
made by policymakers and other leaders in the field (see, e.g., Lain, 2012; MDRC, 2013). 
Some of these reforms have involved technology. The Tennessee Board of Regents, for 
example, provided grants to institutions “to support technology-supported active learning 
strategies aimed at improving student learning outcomes, accelerating time to credit¬ 
bearing courses, and reducing instructional costs” as part of the state’s Developmental 
Studies Redesign initiative (Crandall & Soares, 2015, p. 11). Legislation in Texas that 
mandated developmental education redesign instructed the state’s Higher Education 
Coordinating Board to make provisions “for using technology, to the greatest extent 
practicable consistent with best practices, to provide developmental education to 
students” (Texas S. B. No. 162,2011, as quoted in Texas Higher Education Coordinating 
Board, 2012, p. 28). Recommendations to use technology in developmental education 
reforms have arisen at a time when technology-centered reforms are occurring in a 
number of other policy areas as well, such policing (Spivak, McGough, & Rodriguez, 

2016) , physician education (Han, Resch, & Kovach, 2013), voter registration reform 
(Corley & Palmer, 2015), andK-12 education (National Conference of State Legislatures, 

2017) . 


However, evidence of the effectiveness of technology to improve educational 
outcomes is mixed. While some studies indicate that technology may enhance student 
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achievement (Testone, 2005), other experiences with technology have resulted in no such 
improvement (Jacobson, 2006). Initial research indicates that students tend to complete 
online courses at lower rates and earn lower grades than their peers in face-to-face courses 
(Bailey, Jaggars, & Jenkins, 2015). Other studies demonstrate that underprepared college 
students tend to struggle in entirely online classes (Smart & Saxon, 2016; Xu & Jaggars, 
2014), and that greater structure and classroom practices help students to complete 
computer-mediated courses (Fay, 2017). 

In light of the mixed evidence regarding the effectiveness of technology to 
improve developmental education, it is important to consider how institutional actors 
make decisions about using technology in these programs, and currently, little is known 
about this. Some argue that decisions about technology use in educational settings should 
be made carefully and, ideally, based on evidence that such technologies will enhance 
student learning (Ashford-Rowe & Holt, 2011; Fuhrman, 2017). Indeed, some studies 
have found that evidence-supported education reforms are related to increased student 
achievement (see, e.g., Carlson, Borman, & Robinson, 2011; Van Geel, Keuning, 
Visscher, & Fox, 2016). Evidence that organizations rely upon when making decisions 
may range from the findings of rigorous, systematic studies used in “data-driven decision¬ 
making” (Swan, 2009) to information gleaned from decision makers’ own past 
experiences. Hollands and Escueta (2017) report that higher education leaders often rely 
on local knowledge and information, while relatively few such decision makers cite 
academic research. Moreover, factors unrelated to best practices or evidence of 
effectiveness sometimes drive decisions about educational technology (Ribeiro, 2016). 
Given the financial investments and infrastructure requirements that new technologies 
often involve (Carbonell, Daley-Hebert, & Gijselaers, 2013; Porter, Graham, Spring, & 
Welch, 2014), as well as the need to improve academic success for developmental 
education students, it is important to know whether effectiveness of technology is a 
driving factor for institutional decision makers when innovating with instructional and 
student support technologies. Also, in light of calls by policymakers for increased use of 
technology in developmental education, it is useful for institutional leaders to understand 
the challenges associated with implementing educational technologies. 
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3. Methodology 


This research is part of a larger descriptive study, conducted through CAPR, 
whose purpose is to develop a better understanding of the practices and techniques that 
higher education institutions use to provide developmental education. In light of calls to 
reform developmental education using technology, an important component of this study 
is to analyze how institutions use technology to provide developmental education and to 
foster student success in such programs. The research questions addressed in this paper 
are: 


1. What have been the experiences of higher education organizations with 
the use of technology in developmental education programming? 
Specifically: 

a. What types of technology are used to provide developmental 
education instruction and other student support? 

b. What challenges have institutions faced with regard to the use of 
technology in developmental education? 

2. What considerations have influenced decisions regarding technology 
use in developmental education? 


Data Sources and Data Collection 

Data pertaining to the issue of technology used in developmental education are 
drawn from the qualitative component of the CAPR descriptive study. 1 We conducted 
telephone interviews with key institutional personnel at public open-access two-year and 
broad-access 2 four-year institutions across the United States (“institutional sample”), as 
well as with individuals working in higher education systems, state departments of 
education, statewide associations representing higher education institutions, and 
organizations whose work involves overseeing, administering, or organizing multiple 
two-year and/or broad-access four-year campuses within a state or jurisdiction (“state- 
level sample”). About one quarter of the organizations from which we interviewed were 

1 Another component of the CAPR descriptive study involves a nationwide survey about 
developmental education practices. The survey results are not part of the analysis presented in this paper. 

2 For purposes of this study, a “broad-access” institution is one that has admitted at least 70 percent of 
applicants according to the Integrated Postsecondary Education Data System (IPEDS) database. All public 
two-year colleges are therefore broad-access institutions. Indeed, two-year public colleges, or community 
colleges, are typically referred to as “open-access” institutions as they enroll virtually any student who 
wants to attend. We also note that some community colleges in our sample confer bachelor’s degrees as 
well associate degrees and various certificates. For simplicity’s sake, we refer to such institutions as two- 
year public colleges or community colleges, terms we use interchangeably. 


5 



purposefully sampled from institutions and state-level organizations in five particular 
states — California, Florida, New York, Tennessee, and Texas. Four of these states 
(Florida, New York, Tennessee, and Texas) were selected for particular attention because 
they have been engaged in developmental education reform and/or analysis (see, e.g., 
Dougherty et al., 2016; Lehr & McColskey, 2013; Tennessee Higher Education 
Commission, 2011), and California was selected because it possesses more public 
institutions of higher education than any other state (see Digest of Education Statistics, 
2016). The remainder of the organizations in our sample were selected from a random 
national sample of broad-access public institutions and state-level organizations 
representing or overseeing at least one broad-access institution. 3 

Table 1 presents the number of individuals and organizations represented in each 
of our samples. As shown, we interviewed a total of 127 individuals across 83 
organizations. 4 Organizations located in 36 states and one additional U.S. jurisdiction are 
represented in our sample. Respondents were faculty or administrators who had some 
responsibility for developmental education within their institution or system. During the 
interviews, we asked respondents about of variety of topics relating to developmental 
education, including assessment and placement practices, instructional practices, faculty 
engagement in developmental education reforms, and technology use in developmental 
education programming. With regard to technology, we asked whether the respondent’s 
organization required or encouraged faculty to integrate technology into developmental 
instruction, and if so, what type of technology is typically used. Respondents also 
occasionally discussed technology in response to other questions, such as when 
describing accelerated or modularized developmental education programs. When 
respondents’ statements were vague or the researchers sought to supplement respondents’ 
statements with additional information, we also reviewed relevant documents, such as 
information posted on the websites of state- or system-level entities, higher education 
institutions, or other organizations, as well as relevant published literature. 


3 These institutions were drawn from a list of broad-access institutions surveyed for the CAPR 
descriptive study’s national survey, which asked institutions around the country about their developmental 
education assessment and instructional practices. A total of one hundred public institutions were drawn, 
including 71 two-year public institutions and 29 four-year public institutions. (The relative numbers of 
institutions drawn matched the proportions of two- and four-year public institutions in the overall survey 
population.) Among these institutions, 26 two-year and 6 four-year institutions participated in the 
interviews. The state-level organizations in our sample came from a list developed by the research team of 
state-level organizations overseeing, representing, or coordinating at least one broad-access higher 
education institution. 

4 When multiple individuals within a single organization were interviewed, they were sometimes 
interviewed together and sometimes interviewed separately. 
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Table 1 


Institutional and State-Level Samples, by Organizational Type 



Institutional Sample 

State-Level Sample 

Sample Type 

Open-Access Broad-Access 

Two-Y ear F our-Y ear 

Public Colleges Public Colleges 

Open-Access Broad-Access 

Two-Y ear Four-Y ear 

Public Colleges Public Colleges 
Only Only 

Both 

Purposeful 

5 organizations, 5 organizations, 

6 respondents 6 respondents 

1 organization, 4 organizations, 

2 respondents 6 respondents 

5 organizations, 

6 respondents 

Random 

26 organizations, 6 organizations, 
43 respondents 9 respondents 

11 organizations, 8 organizations, 

18 respondents 16 respondents 

12 organizations, 
15 respondents 

Total by 

organization 

type 

31 organizations, 11 organizations, 
49 respondents 15 respondents 

12 organizations, 12 organizations, 
20 respondents 22 respondents 

17 organizations, 
21 respondents 

Total by 
sample 

42 organizations, 64 respondents 

41 organizations, 63 respondents 

Total 

83 organizations, 127 respondents 



Data Analysis 

With interviewees’ consent, we audio recorded all but six respondents’ 
interviews. These audio recordings were transcribed, and the transcripts were uploaded 
to our database within the Dedoose qualitative data analysis platform. In the few instances 
in which interviews were not recorded, interviewers’ notes were used in place of verbatim 
transcripts. Members of the research team then used a standard coding scheme to code 
each transcript. The coding scheme was developed by the CAPR descriptive study 
research team based on the study’s research questions, a review of relevant literature, and 
themes and patterns that emerged from an initial review of interviews. Some emergent 
patterns (see, e.g., Merriam & Tisdell, 2016) included issues relating to decision-making 
and challenges related to technology use. Data analysis revealed that respondents 
representing more than 90 percent of the organizations in our sample discussed 
technology use in developmental education. Respondents representing just over half of 
the organizations in our sample discussed some aspect of the reasoning behind 
organizational decisions, and respondents representing just under half of the 
organizations in our sample discussed challenges regarding technology use in 
developmental education. 
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After all interview data were coded, we exported the transcript data identified as 
relating to technology use into a spreadsheet that identified respondents’ position (e.g., 
English faculty, mathematics faculty, administrator) and organizational type (e.g., two- 
year institution, four-year institution, state-level organization) for each piece of coded 
data. We then reviewed the coded data to identify emerging patterns and themes (Merriam 
& Tisdell, 2016). Regarding issues about decision-making and challenges associated with 
technology use in developmental education, we conducted an inter-rater reliability 
exercise (Armstrong, Gosling, Weinman, & Marteau, 1997), in which two researchers 
independently coded the data and compared application and consistency of codes. This 
exercise resulted in a small number of minor coding disagreements, which the raters 
discussed with a third researcher in order to arrive at a coding consensus. From there, 
broader themes were identified in coded data and categories representing these themes 
were consolidated in data reduction tables. 
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4. Results 


Technology Use in Developmental Education 

Types of Technology Used 

Institutional and state-level respondents described uses of technology in 
developmental education that can be divided into three categories based on the 
technology’s function: instructional technology, course management technology, and 
student support technology. Table 2 depicts this typology of technology use and describes 
how each type of technology has been used in developmental education. Most of our 
findings focus on instructional technologies. 


Table 2 


Types of Technology Used in Postsecondary Developmental Education 




Type of Technology 



Instructional Technology 

Course Management 
Technology 

Student Support 
Technology 

Definition 

Technology that provides 
instructional content to 
students (e.g., reading, 
writing, or mathematics) 

Technology that organizes 
course materials and 
makes them electronically 
available to students 

Technology that provides 
extracurricular assistance 
to students through 
electronic methods 

Examples of brands 

ALEKS, Cengage, 

Hawkes Learning, 

Pearson “Labs,” Weaver, 
Khan Academy 

Blackboard, Canvas, 
Moodle, WebCT, 
Desire2Leam 

Pearson Smarthinking, 
Starfish 

Examples of use 
in developmental 
education 

• Provides subject 
matter information, 
assessments, and 
assignments 

• Online video lectures 

• Open educational 
resources and other 
electronic course 
materials 

• Provides electronic 
storage of and access to 
course materials, online 
assessments, student 
grades, class-wide 
communications, 
instructional videos, 
course evaluations, and 
discussion boards 

• Primary course delivery 
system for online 
courses 

• Asynchronous online 
tutoring/remote access 
to tutors 

• Electronic early 
warning systems 
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Instructional technology, with regard to developmental education, refers to 
software and other technologies that provide the instructional content of the course — be 
it reading, writing, or mathematics — to students. Frequently reported examples of such 
technology include Pearson “Lab” software packages and other similar products, such as 
Flawkes Learning or ALEKS (for math) or Weaver (for reading). Pearson’s MyMathLab, 
for example, provides “multimedia e-text, video lectures, computational examples, 
animations, interactive tutorials, practice exercises, and sample quizzes and tests” (Raines 
& Clark, 2016, pp. 24-25). Instructional technology also includes other electronic 
methods of providing subject matter information to students, such as online video lectures 
or electronic textbooks (see EDUCAUSE, 2010; Thompson, 2011). This type of 
technology can be used across different types of course formats. Such technologies have 
been used in a variety of ways by institutions in our sample, both before students enroll 
in classes and after enrollment. During the pre-enrollment period, instructional 
technologies have been used to prepare prospective students for assessments of their 
college-readiness skills. These programs were sometimes offered as online courses. Other 
times, instructional software programs such as Pearson “Labs” were provided to pre¬ 
college students (for example, through local high schools), to provide students with 
developmental instruction before they reach the postsecondary level. Instructional 
software has also been used in pre-enrollment bridge programs or “boot camps,” often 
taking place over the summer, before students begin the academic semester. These 
technologies are used in such programming to improve students’ English, reading, or 
mathematics skills with the goal of enabling them to improve their scores on readiness 
assessments and enroll in college-level work by the time the semester begins 
(Edgecombe, Cormier, Bickerstaff, & Barragan, 2013). 

Once students had enrolled in developmental courses, instructional technology 
was used across different kinds of course delivery methods. For example, respondents 
told us of students using instructional software to complete assignments as part of their 
coursework, which occurred in both lecture-based classes and emporium-style courses, 
in which class takes place in a computer lab and instructors provide assistance as students 
work on individualized computerized lessons (see Twigg, 2003). Additionally, some 
institutions in our sample have offered mathematics computer labs that employ 
instructional software applications to provide additional assistance to students outside of 
class time. Instructional technology also includes the use of open educational resources, 
which are teaching and learning resources such as books or other course materials that 
are free or low-cost and available electronically (EDUCAUSE, 2010). Other electronic 
textbooks, such as those produced by Cengage, were also described by our respondents. 
Online video lectures also fall into the instructional technology category. These include 
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online videos produced by Khan Academy, a web-based provider of short instructional 
videos in a variety of subject areas (Thompson, 2011). 5 

Course management technology (sometimes referred to as a “learning 
management system” or LMS) involves using technology to organize and present course 
structure and materials. Some specific uses for course management technologies are 
electronic storage of and access to important course materials (such as the syllabus, 
required reading materials, and lecture presentations), online quizzes and other 
assessments, student grades, class-wide communications (such as emails or electronic 
announcements), a course calendar (often providing deadlines for assignments), links to 
instructional videos, electronic course evaluations, and online discussion boards (see 
Dalsgaard, 2006; Qutab, Shafi-Ullah, Safdar, & Khan, 2016; Wemet, Olliges, & Delicath, 
2000). Some brands of learning management systems include Blackboard, Canvas, 
Moodle, WebCT, and Desire2Learn (authors’ interviews; Dalsgaard, 2006; Qutab et al., 
2016; Wemet et al., 2000). Like instructional technology, course management technology 
can be used across different course formats. Such technology would conceivably be more 
critical in courses that provide all or part of their delivery online, but our respondents also 
discussed instructors in traditional, face-to-face classes using these systems to make 
course materials electronically available. Some respondents indicated that their 
organizations provide a learning management system for all courses. The scheduling 
aspect of course management technology might be particularly useful to students in 
developmental education, who may lack self-regulation skills (as some of our respondents 
suggested) and therefore stand to benefit from the structure and notice of deadlines that 
course management technology provides. 6 

Student support technology is the use of technology to support students’ academic 
performance either by providing individualized assistance with academic tasks or by 
monitoring students’ academic behavior (such as course attendance and performance) to 
ensure they are staying on track to complete their courses. These electronic services 


5 We also note that instructional technology appears to be more widely employed in developmental 
mathematics than in developmental reading and writing among the organizations in our sample. While 
emporium-style classes, extracurricular academic support for mathematics in computer labs, and 
entirely online courses have been used to teach developmental math, respondents mentioned fewer 
instances of such extensive technology use in the teaching or tutoring specifically of developmental 
English. This does not imply that technology is not widely used in some fashion in developmental 
English instruction. To the contrary, many of our respondents identified various ways that technology 
has been used in teaching developmental English, as described in the sections above. The use of 
technology to teach mathematics, however, was more central to course delivery and reported more 
frequently among the organizations in our sample. 

6 Wemet et al. (2000) found that older-than-traditional students identified the “course calendar” aspects 
of a learning management system as instrumental to their involvement and enhanced performance in a 
course (p. 500). Additional research is needed to determine if the same applies to underprepared students. 
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include online access to remote tutors to assist students with academics and learning 
(authors’ interviews; Britto & Rush, 2013; Price, Richardson, & Jelfs, 2007). A brand of 
online tutoring that was mentioned multiple times by our respondents was Smarthinking, 
a Pearson product. Online tutoring provides as much as 24-hour remote academic 
assistance via the Internet from tutors in a variety of subject areas. One respondent 
described electronic tutoring as being “asynchronous,” meaning that students and tutors 
need not be online at the exact same time. 

Respondents also described the use of electronic “early alert” systems to identify 
students who are in danger of failing and targeting those students for extra advising or 
other assistance (authors’ interviews; Britto & Rush, 2013; Faulconer, Geissler, 
Majewski, & Trifilo, 2014). These products track students by monitoring their academic 
behaviors and activity on learning management systems, and then notifying advisors if a 
student is not fulfilling course requirements. Such software can trigger an intervention by 
an advisor or other individual at the college to help prevent students from failing classes 
or dropping out of their programs (Britto & Rush, 2013; Faulconer et al., 2014). 

Challenges Encountered When Using Technology in Developmental 
Education 

When describing various types of technology used in developmental education, 
just under half of our respondents also described challenges they or their institutions 
experienced when implementing those technologies. 7 These challenges can be 
categorized as follows (in order of reported frequency): end-user difficulties, cost-related 
challenges, product limitations, and unavailability of technology. Although many of these 
challenges may not be particular to developmental education (cost-related challenges, for 
example, are issues for all of higher education), end-user difficulties that pertain to 
students and product limitations may be particularly relevant in the developmental 
education context. 

End-User Difficulties 

The challenge that was described by our respondents more frequently than any 
other (described at just under 45 percent of organizations whose representatives discussed 
challenges) related to the difficulty that end users, including both students and faculty, 


7 Issues relating to challenges with technology use was a finding that emerged in the process of data 
collection; therefore, not all of our respondents discussed it. In this section, when we describe the proportion 
of “respondents who discussed challenges,’’ this refers to the proportion of only those respondents who 
described technological challenges in their interviews, which was respondents representing a little less than 
one half of the organizations in our total interview sample. 
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have experienced with technology. A community college system-level respondent stated, 
“Some students will do all right with online education for ... developmental subject 
material. Others are not going to thrive in that environment.” Statements indicating that 
students experienced difficulty using instructional technology were made by respondents 
representing just under a quarter of organizations in our sample for which technological 
challenges were described. Some institutional personnel believed that students sometimes 
lack sufficient skills to use technology in an effective manner. For example, one 
respondent at a community college heard reports of students lacking keyboarding skills, 
which are important for using a personal computer. Additionally, some faculty, like 
students, do not possess the requisite skills to use technology effectively. As one 
administrator at a community college stated, “There are some faculty members who don’t 
have the technology skills themselves necessarily to really feel successful in [technology 
use].” 


More than one sixth of organizations in our sample whose representatives 
discussed challenges noted that end users sometimes had a difficult time with the 
reduction (and in some cases outright removal) of in-person interactions as a result of 
using technology in instruction. These respondents viewed face-to-face interaction as 
beneficial to the learning process and noted that the implementation of computer- 
mediated instruction diminishes the opportunity for in-person interactions. For example, 
a two-year college administrator complained that instructional software had changed the 
in-class dynamic, saying, “rather than being a supplement, it kind of turned into the 
course,” and that “the instructor is still willing to answer questions and help out, but I 
think that students were more hesitant... to ask for that help.” Other respondents echoed 
the perspective that the removal of in-person interaction was detrimental to the 
educational process and posed difficulty for at least some students at their institutions. 

Insufficient Resources 

Respondents at about one fourth of the organizations in our sample for which 
challenges were discussed indicated that costs or a lack of sufficient resources at their 
organizations created challenges when it came to implementing technology in 
developmental education. Challenges related to costs sometimes meant that an institution 
would choose one option over another, or decide to forgo a particular kind of technology 
that was deemed too expensive. Other times, respondents explained that a technology was 
expensive, but the institution chose to use it anyway because it was a superior product or 
met a particular organizational need. Respondents also discussed the fact that institutions 
incur ancillary costs linked to the increased demand for technology in instruction, such 
as costs associated with the use of computer labs or additional training for faculty. As an 
administrator at a community college system said: 
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I think our biggest challenge is the professional development that we need 
to make available for our faculty. Because when you talk about resources 
and you include funding in that conversation, if funding is shrinking ... 
typically one of the areas that gets hit is the professional development 
funding. 

Product-Specific Limitations 

About a fifth of the organizations in our sample whose representatives discussed 
technological challenges reported that particular products were limited in what they could 
do. Some respondents complained that certain instructional software products did not 
provide a useful tool for integrated reading and writing classes, when both of these 
subjects are taught in the same course (see Bickerstaff & Raufman, 2017, and Perin, 
Raufman, & Kalamkarian, 2015, regarding integrated reading and writing). Also reported 
was the perspective that a software vendor may have an excellent product for 
developmental math but not a similarly useful tool for developmental English. Other 
respondents made general statements about software products’ limitations — saying that 
a product “hasn't been without its hiccups” or that it is “not a perfect software by any 
means” — but also indicated that these limitations were not preventing the institution 
from using the software in developmental education programming. 

Unavailability of Technology 

Another challenge reported multiple times (discussed by respondents representing 
about 15 percent of the organizations for which challenges relating to technology were 
discussed) was that crucial technology is not always available, particularly for low- 
income students and students or institutions located in rural areas. One community 
college student services administrator told us that “very few of our students actually have 
Internet outside of class” and noted that the instructional software product being 
discussed “just wasn’t effective for those students.” Technology availability is also a 
problem when institutions or students experience technical difficulties, Internet outages, 
or loss of access codes, all of which result in the unavailability (even if only temporarily) 
of educational technologies. 

These Challenges Align With the Literature 

Our findings about challenges associated with technology use in developmental 
education largely align with challenges related to technology use in education more 
broadly as described in the literature. Resource and cost concerns, availability of 
technology (including technical difficulties and software problems as well as Internet 
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unavailability for low-income students), and end-user problems (such as the need for end 
users to develop new skills and the lack of technical skills among some students) have all 
been documented in the literature as challenges to using technology in education (see, 
e.g., Bell & Federman, 2013; Butler & Sellbom, 2002; Groff & Mouza, 2008; Harrington, 
2010). 8 Groff and Mouza (2008) also identify “lack of human support and infrastructure” 
as a school-level challenge with educational technology use (p. 24), which echoes our 
finding about the removal of in-person interaction posing a challenge for some end users. 


8 From a different perspective, Karp and Fletcher (2014) note that factors such as the provision of 
training for end users, the selection of appropriate technologies, and the development of sufficient 
infrastructure can facilitate successful implementation of student services technology in higher education 
institutions. These steps could be taken to prevent or address challenges using technology such as the ones 
revealed by our research. 
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Reasons for Organizational Decision-Making About Using 
Technology 

Decisions about the use of technology in developmental education have been 
based on a variety of considerations. According to our respondents who discussed their 
reasoning, some decisions were based on economic considerations, while others were 
motivated by a desire for the personalized educational experience that technology is 
perceived to offer. Although expectations about technology’s effectiveness for improving 
student outcomes was among the reasons given (reported at a little over a quarter of the 
organizations for which decision-making factors were discussed), product-specific 
characteristics (reported at just under a quarter of these organizations) and state 
government influence (reported at about a quarter of these organizations) were each cited 
as reasons for technology use almost as frequently. Further, respondents representing just 
over two fifths of the organizations for which decision-making factors were discussed 
cited economic considerations as an influence on technology decisions — more than any 
other single influence. Some respondents discussed making decisions about whether to 
use technology at all, while others discussed reasons for selecting one particular 
technology rather than others. Moreover, some decision-making factors are related to 
others, and some decisions are made based on multiple considerations instead of just one. 
Nonetheless, this section separately considers the reasons given by our respondents, so 
that we can examine each decision-making factor individually. The considerations 
underlying decisions regarding technology use in developmental education uncovered by 
this study are summarized in Table 3 and are discussed in more detail in the subsections 
that follow, which are presented in the order of frequency with which each consideration 
was reported to us. 
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Table 3 


Considerations Underlying Decision-Making Regarding Technology Use in 

Developmental Education 

Examples 

• Choosing a technology because it is low- 
cost or forgoing a technology because it 
is expensive 

• Adopting technology to save money 

• Receiving grant binding or state 
appropriations to implement a particular 
technology 

Effectiveness of technology for educational outcomes • Awareness of evidence demonstrating 

technology’s effectiveness or 
ineffectiveness for improving educational 
outcomes 

• Decision maker’s previous experience 
with a technology’s demonstrated 
effectiveness or ineffectiveness in 
improving educational outcomes 

State- or system-level influence • Policy mandate 

• State- or system-wide license to use 
particular technology 

• State or system demonstrates its 
prioritization of educational technology 

Product or vendor characteristics • Customer service or training provided 

• Sales pitch 

• Feature of the technology meets an 
organizational need 

• Technology lacks characteristics that the 
organization needs 

Personalized educational experience • Technology more easily allows for more 

instruction to be tailored to individual 
student needs 

• Modularized technology allows students 
to complete coursework at their own pace 

Ubiquity of technology • Technology is increasingly pervasive in 

society, which raises expectations that it 
will be used in higher education 

• Organizational culture favors technology 


Decision-Making Consideration 
Economic considerations 
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Economic Considerations 

Participants identified considerations relating to costs, resources, and economics 
more than any other reason for technology decisions (reported at just over two fifths of 
organizations for which decision-making about technology was described). Some 
decision makers selected a particular technology because it cost less than alternatives or 
chose to forgo a particular technology if it cost too much. A system-level administrator’s 
statement indicated that a factor behind the selection of a particular brand of technology 
was this: “It’s a dollar a student based on [full-time equivalency]. ... I don't know about 
any other technology that is that inexpensive for a campus — a dollar a student — oh my 
gosh. That’s incredible!” In another case, an English faculty member explained that a 
particular brand of instructional software was piloted for developmental education, but 
“was too expensive, so we did not use [it] again.” One community college faculty member 
informed us of a situation in which an institution chose to implement an entirely 
technology-based instructional method because the college lacked sufficient resources 
not to do so. This respondent said that small colleges in the state transitioned to a 
technology-based modular approach to developmental math following a statewide 
requirement to reform developmental education, but did not retain elements of face-to- 
face instruction alongside the online modules because the colleges “didn’t have the 
resources to continue with the traditional lecture for students who might benefit from 
that.” 


Institutions also considered costs for students when making decisions about 
technology. For example, one community college mathematics professor explained that 
his college allows students to access online textbooks rather than purchase physical 
copies in order “to save students money.” Another respondent, who spoke on behalf of a 
university system, told us that electronic open educational resources are being used “to 
mitigate costs for students with regard to the $200 textbook.” A different system-level 
respondent reported “a real push” for institutions to use open educational resources — 
particularly in the community college sector — because “the cost of course materials is a 
constant issue.” 

The receipt of grant funding was cited as at least one of the reasons behind using 
technology in several cases. A couple of institutions reported receiving grants pursuant 
to Title III of the Higher Education Act, which they used to fund technology initiatives 
relating to developmental education. Title III grants award discretionary funding to 
eligible colleges and universities for the purpose of enhancing certain aspects of their 
operations, including faculty training, developmental education, and other activities that 
“improve and strengthen [their] academic quality, institutional management, and fiscal 
stability” (U.S. Department of Education, n.d.). Institutions in our sample used Title III 
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funding to provide (among other things) faculty professional development with online 
technology and tutoring in computer labs for developmental math students. Other 
institutions received funding from private foundations or from state legislatures or 
systems for purposes of developing technological innovations related to developmental 
education. 

Effectiveness of Technology 

A number of our respondents (representing a little over a quarter of organizations 
that reported decision-making factors) indicated that decision makers’ understanding 
about a given technology’s effectiveness (or lack thereof) to improve educational 
outcomes played a role in the organization’s reasoning for technology use in 
developmental education. Often, perceptions of effectiveness were expressed by our 
respondents in somewhat vague terms, with some citing general “evidence” or what they 
“believe” to be effective for developmental education students. Despite the fact that these 
decisions were not necessarily based on specific systematic research, effectiveness 
considerations played a role in these organizations’ decisions about whether and how to 
use technology. For example, a community college representative expressed a general 
awareness that students “struggle” with entirely online classes, and because of this, the 
institution was looking into how to assess new students’ “qualifications” to ensure that 
they are “really ready to take that distance [education] course.” This illustrates how 
institutional decision makers may have a basic impression about technology’s 
effectiveness, even if they do not have strong evidence of it. 

We were also told of instances in which decision makers’ own previous 
experiences with technology provided evidence of effectiveness and therefore influenced 
future decision-making. In one example, a community college administrator spoke about 
the college’s plan to expand a digital textbook initiative that began with developmental 
education: 


We’ve done some pilots and we’ve focused those specifically with our 
developmental courses. ... That was really, I would say, one main impetus 
in moving forward with the [initiative], because the retention rates as well 
as the success rates for those students were significant, so we really 
wanted to move that college-wide. 

A combination of previous experience and knowledge of relevant research may 
also inform organizational decisions about the use of technology in developmental 
education. This combination of factors was illustrated by a university administrator 
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discussing the fact that the institution had instructional technology “available” but that it 
was “not utilized that much” for developmental education students: 

I have found that developmental students ... don’t function well with 
[computer programs]. ... Research has shown that developmental 
students in general do not respond that well to computer online programs. 

... And I’ve seen it for about the 20 years that I’ve been teaching 
developmental education. 

State- or System-Level Influence 

A similar number of respondents who reported effectiveness as a decision-making 
consideration also reported state- or system-level influence in decisions to adopt 
technology in developmental education (about one quarter of the organizations for which 
decision-making about technology was discussed). Sometimes, a statewide mandate to 
redesign or to integrate technology in developmental education led institutions to adopt 
instructional technologies. For example, a system-level administrator explained that 
under that state’s policy mandate, “one of the required components of developmental 
[education] programs is the integration of technology with an emphasis on instructional 
support programs.” 

State-level actors can also influence decisions about technology in less direct 
ways. The existence of a statewide license to use a particular product or brand of 
technology has led decision makers to employ that particular technology at institutions 
within the state. Additionally, states can influence institutional technology decisions by 
demonstrating that the state prioritizes technology and by providing resources to 
campuses for adopting new technologies. One system-level respondent said that a state 
higher education board had created an office dedicated to educational technologies and 
provided online faculty training somewhat frequently, and that these actions have “really 
helped to push out the innovation around technology.” A different system-level 
respondent said that some high-ranking academic officers were “asking what technology 
they are currently using and looking to see if there is a way to ... share best practices 
across the system.” This respondent viewed such activities as sending a system-wide 
“message ... that the use of technology in a variety of ways can really help students be 
successful.” 

Product or Vendor Characteristics 

Sometimes organizations selected a particular product or brand of technology 
because of characteristics unique to the product or vendor. Such considerations were 
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reported at just under a quarter of the organizations in our sample for which decision¬ 
making about technology was discussed, slightly less frequently than either effectiveness 
or state/system-level influence. Multiple respondents cited a particular vendor’s customer 
service or “sales pitch” as a reason for choosing a type of technology. A relatively unique 
feature of one type of technology was cited as a reason for deciding to use it when that 
unique feature met an institutional need. An example of this is an institution selecting a 
software product that provides integrated reading and writing material — something that, 
as of the time of our interviews, not a lot of software vendors provided. Relatedly, a few 
respondents identified particular shortcomings with technology products as reasons they 
were not being used. As noted when discussing challenges associated with technology 
adoption, a system-level administrator said that a certain software vendor’s exclusive 
focus on mathematics and not on English was a reason that the product was not used more 
frequently at institutions within the system. 

Personalized Education 

Respondents at about one fifth of the organizations whose representatives 
discussed decision-making identified the desire for personalization in instruction as a 
motivation for employing technology in developmental education. These respondents 
seemed to believe that technology can be used to tailor the curriculum to a student’s 
individual educational needs, and that this personalization would enhance the educational 
experience. One way that technology can personalize education involves the use of 
diagnostic components of certain instructional software products (such as ALEKS or 
Pearson “Lab” products) to inform instruction. In the words of one community college 
administrator, these assessments help educators “to see where the students are, where the 
gains have happened, and where gaps [in student mastery] still exist.” Instruction that 
takes place in computer labs (as with emporium classes or “math learning labs”) was also 
thought to provide an individualized approach. A representative of a community college 
system explained that students “go into the lab and it’s individualized supplemental 
instruction. ... Lab students are there doing individualized work [and the] faculty member 
is going around and helping them throughout the time they’re there.” 

Personalized education also includes using technology to modularize course 
content and allow students to complete each module at their own pace. A community 
college faculty member, discussing a decision made at the system about modular math, 
noted that the modules were “technology-based, for the most part” and stated: 

[It] didn’t feel like students needed that entire course, so they created 

different exit points for different programs of study. ... There is now a 
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four-week interval course which allows students to finish their 
requirements a lot sooner. 

Similar perspectives were discussed by other respondents with regard to 
computer-mediated self-paced developmental education modules. It should be noted that 
while self-paced courses theoretically enable students to complete requirements at a faster 
pace, this does not always occur in practice. For example, Bickerstaff, Fay, and Trimble 
(2016) found that in some self-paced courses, students have made “less progress than 
intended by the design of the course” (p. 24). Nonetheless, the personalized nature of self¬ 
pacing was discussed by multiple respondents as reasoning for the use of computer- 
mediated self-paced courses in their curricula. 

Ubiquity of Technology 

Although reported less frequently than the previously described considerations, 
several respondents (representing just under one sixth of organizations for which 
decision-making was described) made statements indicating that ubiquity of technology 
influenced decision-making about whether and how to use technology in developmental 
education. These statements suggested that because technology is so pervasive in 
contemporary society, it is expected that higher education institutions will use technology 
in some way, and therefore organizations feel they must accommodate and use 
technology in — among other things — developmental education. One administrator at a 
university system said that technology integration in developmental instruction was not 
“required” at institutions within the system, but that “because of the way curricula [have] 
developed over the years and the way faculty approach their courses ... certainly 
technology is playing a bigger and bigger role.” A community college administrator 
discussed reasoning for using technology to teach developmental writing, which 
illustrates how increased ubiquity of technology can influence decision-making about 
using technology in instruction: “When I first came here in the early 90s, I computerized 
our writing courses on the theory that nobody is writing without a computer.” 

Technology may also be so commonly used across a campus or system that it 
becomes part of the organizational culture, or at least prioritized in the minds of decision 
makers. As explained by a student services administrator at a “mostly online” community 
college, “We’re ... a very technology-heavy college normally, so it’s just part of our 
culture. ... We’re always looking for innovative ways to offer online instruction.” An 
administrator at another community college indicated that the institution had invested in 
technology “many years ago,” and that the institution “certainly encourage[s]” 
technology use in developmental education. Statements such as these indicate that the 
ubiquity of technology across an organization as well as across society can influence 
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decision-making about the use of technology in instruction. Such statements also suggest 
that some organizations are (or hope to be viewed as) innovative, and that technology- 
based instruction is one method to help achieve this objective. Moreover, as discussed 
above, state- or system-level mandates influence campus decisions about technology in 
developmental education, and one way that a particular technology can become 
ubiquitous across a state or system is through the imposition of such a mandate. 9 


9 This latter point is an example of multiple or combined influences on decision-making about 
technology use in developmental education. 
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5. Discussion and Conclusion 


This study explores the experiences that institutions of higher education have had 
with technology in the provision of developmental education, including the various types 
of technology used and the challenges faced by institutions when implementing 
technology. Our respondents described technologies that fall under three categories. The 
first is instructional technology, which is content-driven and used to provide reading, 
writing, or mathematics instruction to students either before or after they enroll at an 
institution. The second category is course management technology, which electronically 
organizes and provides online access to course materials, such as syllabi or required 
reading materials, and allows certain course functions such as quizzes and discussions to 
take place online. The third category, student support technology, uses technology to 
assist students with academics or to encourage them to remain on track to complete their 
courses and programs. End users, such as students and faculty, have experienced 
difficulties with using technology, which has presented a challenge for institutions that 
implement technology in developmental education. Other challenges include finding 
funding to cover the costs of technology, lack of availability of technology, and 
limitations of particular products. 

Our analysis of decision-making considerations regarding the use of technology 
in developmental education demonstrates how issues related to costs and resources weigh 
on institutions as they implement new technologies. Among the organizations we studied, 
economic and cost considerations were reported more frequently than any other decision¬ 
making factor. Expectations about the effectiveness of technology for enhancing 
educational outcomes were also considered, but these were reported to us less frequently 
than costs and other economic considerations, and about as frequently as the influence of 
state- or system-level policies and practices. In the field of education, there has been a 
growing focus on the use of data to inform decisions about educational practices (see, 
e.g., Swan, 2009). Data and evidence, however, are not always readily available, and 
other factors — especially costs — may play an outsized role in decisions about 
technology use in developmental education. Moreover, institutional leaders do not always 
have or rely upon strong evidence regarding the effectiveness of technology to improve 
student outcomes; many times, decision makers described somewhat vague perceptions 
of effectiveness when explaining why technology has been used. 

We must also note that the use of educational technology interacts with other 
phenomena at the state and institutional levels, with stakeholders engaged in numerous 
conversations concerning improvements to developmental education as they are making 
decisions about implementing technology-based solutions and reforms. Beyond resource 
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availability, the use of a particular technology depends, at least in part, on whether the 
technology possesses some feature that will meet an organizational need. Furthermore, 
organizational culture with regard to technology, as well as the perception (or reality) that 
such technology is increasingly used and becoming ubiquitous, may also play a role in 
the decision to use technology. 10 Considerations such as these may determine whether, 
how, and in what form technology will be used in an institution’s developmental 
education programming. Technology vendors who persuade decision makers that a given 
product is effective, cost-efficient, and able to meet an organizational need (or some 
combination of these features) may experience greater success in persuading an 
institution to use a particular technology. Indeed, our findings show that a vendor’s “sales 
pitch” has played at least some role in certain organizations’ decisions to use a particular 
technology. 

This study contributes to the literature on developmental education and 
educational technology in meaningful ways. First, we identify the various types of 
technology used in the provision of postsecondary developmental education, and we 
categorize these by their function as instructional, course management, or student support 
technology. Moreover, our findings shed light on some of the factors underlying decisions 
about the use of technology in postsecondary developmental education, illustrating that 
many of these considerations are unrelated to effectiveness in improving educational 
outcomes. 

This study also provides information about challenges faced by those institutions 
implementing technology in developmental education, so that campuses considering a 
new technology may anticipate these challenges and take preliminary steps to prepare for 
them. End-user difficulties with technology — particularly with regard to developmental 
education students, but also faculty — were frequently discussed challenges among our 
sample. This is important because students and faculty should be able to use technology 
appropriately and effectively. Institutions and systems may anticipate this challenge and 
attempt to address it by providing additional training on how to use the technology. Our 
study also provides information to policymakers, institutional leaders, and system-level 
officials on how decisions are made with respect to technology use in developmental 
education. Institutional and state-level leaders who want to encourage evidence-based 
decision-making regarding the effectiveness of educational technology can make efforts 
to identify such evidence, share it with campus stakeholders, and employ it themselves 
when considering the purchase of organization-wide licenses or acquisitions. Leaders 
may also, to the extent possible, provide additional funding for the implementation of 


10 For more about organizational culture and technology use in higher education institutions, see Karp 
and Fletcher (2014). 
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new technologies, so that a potentially effective technology will not be disregarded solely 
based on its cost. 

Underprepared college students require support that will set them up for success 
in their college-level courses and ultimately guide them toward college graduation. 
Student success rates in developmental education are relatively low (Bailey, 2009), and 
the goal of integrating technology in developmental instruction should be to help place 
more students on the path to academic success. This paper has provided information for 
policymakers and higher education leaders to consider when looking for effective ways 
to use technology in developmental education. 
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